Systems and methods for measuring reactions of head, eyes, eyelids and pupils

ABSTRACT

Systems and methods are provided to measure reaction times and/or responses for head, eye, eyelid movements, and/or changes in pupil geometry. The system includes eyewear or headwear including one or more eye-tracking cameras for monitoring the position and geometry of at least one eye and its components of the user, one or more scene cameras for monitoring the user&#39;s surroundings, and one or more processors to determine reaction times. Optionally, the system may include one or more of a multi-axis accelerometer to monitor head movements, light sources to trigger visual evoked responses, and/or electronic inputs that may be used to indicate the time of occurrence of external reference events. Measured reaction times and other measurements may be monitored for use in a range of applications. Responses and reaction times may be measured continuously over extended periods, even over a lifetime to measure consequences of the aging process.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH AND DEVELOPMENT

The U.S. Government may have a paid-up license in this invention and theright in limited circumstances to require the patent owner to licenseothers on reasonable terms as provided for by the terms of Department ofDefense (US Army) Contract No. W81XWH-05-C-0045, U.S. Department ofDefense Congressional Research Initiatives No. W81XWH-06-2-0037 andW81XWH-09-2-0141, and U.S. Department of Transportation CongressionalResearch Initiative Agreement Award No. DTNH 22-05-H-01424.

FIELD OF THE INVENTION

The present invention relates to apparatus and methods for unobtrusivelymeasuring reaction times and/or other responses following events thattake place in the general environment of a device wearer andconsequential changes in the head, eye, eyelid, and/or pupil of thedevice wearer.

BACKGROUND

The response of an individual to a spontaneous or intentionallygenerated event involves a number of complex neurological andphysiological steps including detection, conductive neural pathways,synaptic connections, perception, cognition, and neuro-muscular control.Normal activities or disorders in any of the anatomical structures orchemistries involved in these steps, as well as physiological awarenessand training, can affect an individual's response magnitudes andreaction times. An unobtrusive device that measures reaction timesand/or other responses of an individual over brief (i.e., seconds tominutes) or prolonged (i.e., days or even years) time periods has anumber of medical, sports training, human factors, safety, diagnostic,military, law enforcement, gaming, and other applications.

Applications that involve sensing and machine vision are becomingincreasingly common-place. In part, this has arisen as a result oftechnological advances in the electronics and software developmentindustries, and decreases in the cost of sensors, cameras, andinformation processing units. For example, recent advances inaccelerometers based on microelectromechanical systems (MEMS) techniqueshave made them inexpensive, miniature, sensitive, robust, low-power,multi-axial (within a single package), and easy to use. Accelerometerscan be used to sense gravitational orientation as well asmulti-dimensional movements.

Similarly, cameras that employ complementary metal-oxide semiconductor(CMOS) or charge-coupled device (CCD) approaches, can be inexpensive,miniature, optically sensitive, low-power, robust, and high resolution.Using such cameras and image processing units, automated objectidentification and tracking are increasingly being used in a number ofdiagnostic, human performance, commercial, and control applications.

With the exception of events that cause head trauma, most movements ofthe head are relatively slow (e.g., less than about ten Hertz (10 Hz)).Thus, sample rates to accurately monitor head movement may be in therange of tens of samples/second. Similarly, most measurements of eye,eyelid, and pupillary responses and reaction times, which may havereaction times in the range of tenths of a second to seconds, requirethe frame rates commonly available in modern, household cameras andvideo displays (i.e., 30-120 frames per second). Research laboratoriesand some other applications may demand measurements from accelerometersand cameras that are capable of higher sample rates but at an increasedcost; however, eyewear and headwear devices can take advantage ofcommonly-available miniature, low-power cameras and sensing components.

Many head- and eye-tracking systems use cameras and illuminators thatare located at a considerable distance (e.g., greater than about tencentimeters (10 cm)) from the wearer's head. As the distance away fromthe wearer's head is increased, a head/eye tracking apparatus generallybecomes less obtrusive; however, it becomes increasingly difficult toaccurately measure the location of a wearer's head and/or eyes becauseof the need for higher spatial resolution by cameras. Also, wide-ranginghead movement may cause complete loss of the ability to track thelocation of a wearer's eye and its components.

With the advent of modern-day microelectronics and micro-optics, it ispossible to unobtrusively mount the components for measuring reactiontimes or other responses on eyewear (e.g., eyeglasses frames) orheadwear (e.g., helmet, mask, goggles, virtual reality display)including those devices disclosed in U.S. Pat. No. 6,163,281, 6,542,081,7,488,294, or 7,515,054, the entire disclosures of which are expresslyincorporated by reference herein. The use of low-power and miniaturecameras, sensors, and electronics permits a head-mounted system to benon-tethered through the use (optionally) of a battery power source.Furthermore, recent advances in wireless telecommunications allowreaction time results to be transmitted in real-time to other computing,data storage, or control devices. As a result of these technologicaladvances in a number of fields, an eyewear- or headwear-based responseand reaction time monitoring system may be unobtrusive, light-weight,low-power, portable, and convenient to use.

Non-invasive tools for physiological monitoring and medical diagnosticsare increasingly commonplace in clinics, hospitals, researchlaboratories, and even homes and public areas, such as grocery storesand pharmacies. If such tools are constructed to be unobtrusive, simpleto use, and portable; they gain even further potential in theiracceptance by the general public and subsequent applications to monitorlonger-term response trends (versus one-time or “snap-shot”measurements). This is in addition to their capability to react inreal-time if certain conditions are encountered; for example, if a stateof drowsiness is determined while driving a vehicle.

Monitoring responses of the head and eyes is particularly useful inassessing both central and peripheral nervous system physiologicalfunction and disorders. For example, reaction times and/or otherresponses of the pupil are influenced by a chain of anatomicalstructures including photosensitive ganglion cells and theretinohypothalamic tract within the optic nerve, the pretectal nucleuswithin the upper midbrain, and the Edinger-Westphal nucleus with axonsrunning along left and right oculomotor nerves that synaptically connectto ciliary ganglion nerves that, in turn, innervate constrictor musclesof the iris. Disorders or conditions (e.g., the presence ofbarbiturates) that affect any of the structures within this chain mayproduce consequential changes in reaction times and/or other responsesthat may be monitored non-invasively and unobtrusively. Initialconstriction of the pupils by alcohol or opioids; or dilation by a widerange of drugs including lysergic acid diethylamide (LSD), cocaine,amphetamines, 3,4-methylenedioxymethamphetamine (MDMA, also known asecstasy), mescaline, and so on; may also affect measured responses andreaction times.

Reaction times and/or other responses may be used to probe even deeperinto cognitive functions of the brain. An unobtrusive tool forsubstantially continuously monitoring reaction times and/or otherresponses of the head and eyes may lend valuable quantitativemeasurements to perceptions that we are all familiar with, includingsensing fear, alarm, or whether someone is telling the truth. This mayresult in a number of applications including substantially continuousmonitoring of the elderly for diagnostic purposes as well as an input indetermining emergency situations. An example of an application where aclinician may take advantage of the quantitative nature of suchmeasurements is in the assessment of many anxiety disorders such aspost-traumatic stress disorder (PTSD). Measuring avoidance behaviors anddifficulty concentrating along with exaggerated responses to events thatstartle may be monitored and used for assessment and tracking of suchdisorders.

Measurements of evoked pupillary responses have been used in a widerange of assessments of “cogitative load” or “mental effort” andassociated disorders. Under normal physiological conditions, thepresentation of multiple stimuli in timed sequences can increasepupillary dilation responses. An increase in this effect may beindicative of various forms of dementia.

Conversely, a goal of sports training is to improve performance bydecreasing reaction times. Anticipation and experience may each reducereaction times by (as generally believed) reducing any decision timesand strengthening synaptic connections. An unobtrusive device that maymeasure reaction times and/or other responses over prolonged periods maybe valuable in monitoring, for example, the effectiveness of trainingregimes.

SUMMARY

The present invention is directed to systems, apparatus, and methods fordetermining reaction times and/or other responses of the device wearer.In an exemplary embodiment, the apparatus, systems, and methods hereinmay provide a method and system that measures reaction times and/orother responses, e.g., magnitude of responses, of the head, eye, eyelid,and/or pupil for various applications.

For example, the apparatus, systems, and methods herein relate to theuse of machine vision and other techniques to identify 1) the time ofoccurrence of events that happen in the environment of a device wearer,and/or 2) consequential movements of the head and/or small movements ofat least one eye of the device wearer including eyelid and eyeballdisplacements, and/or changes in the geometry of a pupil, e.g., tomeasure, analyze, and/or determine responses of the wearer. Thedifference between the time of occurrence of environmental events andthe time of the device wearer's response is commonly referred to as a“reaction time” or “response time.” The term “reaction time” isfrequently used to describe consequential responses to events that occurspontaneously. In the case when an eye movement or pupillary response isintentionally evoked using a light source or some other means, the term“response time” is more common in the scientific and medical literature.

Generally, the term “reaction time” will be used herein to refer to thedifference between the time of an event and the time that a reaction tothat event is detected that exceeds a predetermined threshold, e.g.,resulting in a length of time associated with a binary change inreaction. It is to be understood that the use of the term “reactiontime” is intended to include “response time,” “response delay,” “reflextime,” and other equivalent terms.

Similarly, the “magnitude” of a response may also be measured and/oranalyzed in addition to or instead of the “reaction time” of theresponse. Examples of magnitudes of responses include measurements suchas the total spatial displacement of anatomical structures of a deviceuser (e.g., an eye or eyelid), a rotational movement such as thatgenerated by turning the head, the amplitude of a change in velocity oracceleration of anatomical structures, changes in area or volume such asthose experienced during pupillary dilation or constriction, and thelike.

Similarly, in the description below, references to sensing movements ofa device wearer's head also may include any structure or component ofthe head that may simultaneously move and be sensed by eyewear orheadwear when worn by the device wearer. This includes, for example, theskull, brain, hair, ears, forehead, eye sockets, nose, cheeks, chin,jaw, and the like. Along the same lines, references to responses andmovements of the device wearer's eye incorporate all sub-components ofthe eyes including the eyeball, iris, cornea, lens, sclera, ciliarybody, vitreous body, and the like. References to responses and changesof the device wearer's pupil include structures and components that canbe imaged within pupillary images such as the lens, retina, fovea, opticdisk, sphincter pupillae, dilator pupillae, and the like. Similarly,references to movements and changes in the device wearer's eyelidinclude monitoring the eyelashes, eyebrow, folds within the eyelid, andthe like.

A low-cost, unobtrusive, portable platform that may repeatedly measureresponses and reaction times has a wide range of applications. A smallnumber of examples include monitoring the degree of fatigue of anindividual, assessing driver or pilot awareness, assessing the effectsof drugs or alcohol, diagnosing post-traumatic stress disorder, trackinghuman performance with age, determining the effectiveness of training orexercise, controlling various aspects of games, acquiring foundationalclinical data to assess neurological or cognitive disorders, diagnosingand monitoring degenerative eye conditions, and monitoring the awarenessof individuals or animals who are otherwise not able to communicate.Sectors and industries that may make use of reaction time measurementsinclude medicine, military, law enforcement, security, humanperformance, sports medicine, rehabilitation engineering, police,research laboratories, and toys.

In one embodiment, a system for determining reaction times and/or otherresponses includes a device configured to be worn on a person's head, aninput to determine reference times, an eye-tracking camera mounted onthe device and positioned for viewing a first eye of the wearer of thedevice, a processor that determines the locations of anatomicalstructures within eye-tracking camera images, and a processor that usesthe reference times and eye-tracking locations to determine reactiontimes and/or other responses of the wearer's eye, eyelid, and/or pupil.

In some embodiments, reaction times of the device wearer may becalculated by determining the times of occurrence of reference eventsand consequential responses by the device wearer. The apparatus andsystems herein may accept electronic signals to provide temporalreference markers or use one or more scene cameras mounted on eyewear orheadwear to optically identify the times of occurrence of changes withinthe environment. In an exemplary embodiment, the device may include anaccelerometer to measure multi-axis movements of the head, and one ormore unobtrusive eye-tracking cameras may be mounted on the eyewear orheadwear to track movements of, or changes in, one or more eyes of thedevice wearer. Reaction times and/or other response measurements of thedevice wearer may be used in a wide range of fields including medical,legal, security, sports, computer gaming, and human factors.

For example, systems and methods herein may use eye-tracking camerasmounted on eyewear or headwear to track the locations of one or botheyes similar to the embodiments described in the references incorporatedby reference elsewhere herein. Eye-tracking cameras and one or moreassociated processing units may be used to identify and track referencelocations within eye-tracking camera images. Reference locations mayinclude, for example, the center of a pupil, edges of a pupil, center ofthe iris, edges of the iris, and locations of the whites of the eye orsclera. For reaction time measurements, particular attention is paid tothe times when significant movements of reference locations areinitiated.

Systems and methods herein may additionally use eye-tracking camerasmounted on eyewear or headwear to track the locations of one or botheyelids. Eye-tracking cameras and one or more associated processingunits may utilize the obscuring of structures of the eyeball (e.g.,pupil, iris, sclera) and/or changes in color, brightness and/or texturein the region of the eye that result from eyelid closure. In general,absent neuro-muscular disorders, the eyelid sweeps the field-of-view ashorizontal waves of disruption within eye-monitoring camera images. Forreaction or response time measurements, particular attention is paid tothe times when the eyelid initially closes, covering the eye includingthe pupil, and/or as the eyelid re-opens.

Systems and methods herein may additionally use eye-tracking camerasmounted on eyewear or headwear to track the size of one or both pupils.Eye-tracking cameras and one or more associated processing units maydetect the edges of the pupils as regions where there is high spatialcontrast in luminance or texture compared to the iris. Geometric shapes,such as circles or ellipses, may be superimposed on images to determinea size that best overlaps with the edges of the pupil. Areas may then becalculated based on well-known formulae for the geometric shapes.

Additionally or alternatively, the area occupied by a pupil may becalculated by determining the number of pixels that meet certain imagerecognition criteria, such as exceeding a certain darkness level in anarea where there are a significant number of other dark pixels. Thistakes advantage of the fact that, under most lighting conditions, pupilsappear dark and within a certain size range. Areas may then becalculated, for example, based on counts of the number of adjoining darkpixels within a relatively large region of dark pixels.

Additionally or alternatively, with careful control of lighting, eithertranscutaneously from the side of the head or aimed directly (withinabout ±three degrees (3°) of the perpendicular to the surface of thecornea) into the pupil, it is possible to generate a so-called“white-eye,” “red-eye,” or “while-pupil” effect. Under these conditions,it is possible to calculate areas based on the number of bright pixelsand/or by detecting the edges of bright pupil regions.

For reaction or response time measurements, attention may be paid to thetimes when the pupil begins to constrict or dilate. Compared to responsetimes of the eye and eyelid, pupillary responses are generally moredelayed, i.e., with longer reaction or response times.

In order to measure a reaction time, the time of occurrence of an event(i.e., reference time) that is causal to the reaction must bedetermined. There are many potential sources for events that may produceresponses by the head, eye, eyelid, and/or pupil. These may includeimages, sounds, flashes of light, movement, blunt force, and the like.The eyewear or headwear platform is designed to accept different inputsfor reference time measurements from these different modalities.

In accordance with an exemplary embodiment, systems and methods areprovided that use one or more scene cameras to optically monitor theenvironment surrounding the device wearer. Scene cameras and one or moreassociated processing units are used to identify and track referenceobjects within scene camera images. Object recognition based on machinevision techniques, which are well-known in the art, may use identifiablecharacteristics within images of an object such as an object'sbrightness, size, shape, color, and the like; or combinations of thesecharacteristics. The presence or absence of an object within a scene,such as the abrupt display of an image on a computer monitor or therevealing of a photograph, may, for example, be used as a temporalreference.

In accordance with another embodiment, systems and methods are providedthat use the location of one or more objects in scene camera images todetermine reference times. For example, reference times may be triggeredwhen an object enters a certain location or region within scene cameraimages. A sports-related example is the registration of a reference timewhen a projectile, such as a baseball, reaches a selected location(e.g., near a batter) within a scene camera image. Similarly, referencetimes may be registered when an object attains a certain velocity oracceleration within scene camera images. Additionally, reference timesmay be triggered when there is a change in an object's characteristicssuch as brightness level, color, size, shape, and the like in scenecamera images.

In addition or alternatively, one or more scene cameras that survey theenvironment around a device wearer may be used to determine referencetimes by tracking specific objects within scene camera images, and todetermine the occurrence of consequential head movements. In the examplewhere there is movement of one or more scene cameras due to movement ofhead of the device wearer, most of the area within the scene cameraimages appears progressively displaced (i.e., the entire scene shifts).At a constant video frame rate, higher velocity movements of the headresult in greater displacements of the background in scene camera images(except when head movements are directly toward or away from a nearbybackground, such as a wall). This is in contrast to the movement ofsmaller, specific objects within scene camera image that may be used tomake reference time determinations.

In accordance with still another embodiment, an accelerometer may beincluded within a set of eyewear or headwear to monitor head movement.Movements of the head in response to trigger events may be used tomeasure head reaction times and/or other responses. A multi-axisaccelerometer, e.g., that uses MEMS or other techniques to detectaccelerations in up to three (3) orthogonal directions, may be used todetect accelerations of the head as well as overall head orientationrelative to the gravitational field of the earth. Such accelerations maybe integrated with respect to time to obtain relative head velocities ineach dimension as well as head displacements (i.e., following a secondintegration with respect to time). For many applications, it isconvenient to calculate rotational movements of the head (i.e., affixedto the eyewear or headwear containing the accelerometer) about the neck.

In accordance with another embodiment, a general purpose input connectormay be included within the eyewear or headwear to allow a wide range ofsensor signals to be used as a source for reference times. For example,a microphone may be connected and used to generate reference times fromauditory inputs. This allows the device to measure reaction times and/orother responses to the occurrence of sounds. A simple method to registeraudio reference times is to monitor the onset of a sharp sound. Examplesof sources of such sounds include the clapping of hands, an explosion, astarter's gun, a drum beat, a metronome, a click, and the like. Moresophisticated sound-recognition techniques and algorithms are known inthe art including those that recognize the occurrence of specific words,tones, or audio sequences.

In other embodiments, analog signals (i.e., signals represented ascontinuously varying changes in amplitude of either current or voltage)may be used as a source for reference times. For example, these mayinclude sensing the position of an object (e.g., door, lever), intensityof ambient light, temperature, and the like. Temporal references may beregistered when the amplitude of the analog signal exceeds a selectedthreshold or by other waveform recognition techniques.

In other embodiments, digital signals may be used as a source forreference times. In general, the time of occurrence of low-to-highand/or high-to-low transitions, i.e., binary transitions, in suchsignals represent time registrations. Digital signals may, for example,be generated by other control devices such as a computer, produced by aswitch or push button, synchronized with the display of an image,coincident with a tactile or electrical stimulus, and the like.

In accordance with still other embodiments, it is possible to producestimuli directly on the eyewear or headwear in order to evoke measurableresponses. This configuration is particularly useful in measuringpupillary responses and reaction times. For example, light stimuli maybe provided by one or more light-weight and low-power, light-emittingdiodes (LEDs) aimed at one or both eyes. The LEDs and/or other lightsources may be made to vary in wavelength and/or intensity. In additionor alternatively, one or more speakers may be provided, e.g., on thedevice or separate from the device, that may generate predeterminedsounds and subsequent reaction times may be monitored.

Measuring the presence or absence of pupillary responses and reactiontimes may be particularly useful in medical diagnostic screening. Delaysor disruption in pupillary responses may be indicative of a number ofneuro-muscular or cogitative disorders. For example, under normalphysiological conditions, the projection of sufficient light toward oneeye should generate pupillary constriction in the other eye. Absence ordelay of this response may, for example, result from lesions within thecentral nervous system or the presence of certain classes of drugs.

An accurate assessment of reaction times and/or the degree ofresponsiveness may also be particularly useful in law enforcementduring, for example, screening tests for the effects of drugs and/oralcohol. In this application, it is possible to use visible light orlight that is normally “invisible,” i.e., beyond the visible spectrum)to evoke pupillary responses. Visible light, in the wavelength rangefrom about 390 to 750 nanometers, may be used to evoke pupillaryresponses, where the threshold amplitude has been reported to besomewhat wavelength-dependent.

The use of invisible light in the near infrared spectrum, e.g., in thevicinity of 800 nanometers, may also be used to measure evoked pupillarydilations of the device wearer at times that the device wearer is notaware that measurements are being made. This protocol may be used tocircumvent efforts by the device wearer to deceive law enforcement, forexample, by attempting to conceal responses. Evoked responses may also,for example, be generated on the eyewear or headwear; or from anexternal source, such as traditional law-enforcement methods involvingthe sweeping of a flashlight across the region of the eyes, wherereference times of when the flashlight is aimed at the eyes can besensed and registered by either scene or eye-tracking cameras.

An illumination controller may be coupled to one or more light sourcesand may be configured for determining brightness based on the presenceor absence of measurable pupillary responses. By repeatedly adjustingbrightness levels, it is possible to determine threshold levels forevoked pupillary responses, i.e., the minimum change in light level thatcauses pupillary constriction or dilation at a given wavelength. Suchmeasurements may be used in medical diagnosis and physiologicalmonitoring. An unobtrusive eyewear platform may be particularly usefulin cataloging the progression of some forms of degenerative eye diseasesor age over prolonged periods (i.e., months or even years). The sceneprocessing unit and eye-tracking processing unit may be one or moreseparate processors with a separate illumination controller, or may be asingle processor and include the illumination controller.

In one embodiment, the illumination controller may be configured foramplitude modulation of at least the current and/or the voltage to thelight source to provide the desired brightness levels at an eye. Inaddition or alternatively, the controller may be configured forpulse-width modulation of at least one of the current and the voltage tothe light sources to provide the desired brightness levels.

In any of these embodiments; illumination, reaction time measurements,head tracking, eye tracking, and/or gaze tracking may be operatedsubstantially continuously or intermittently. Processors, cameras,illumination, and/or other electronics may be deactivated when not inuse, e.g., to conserve power. Illumination sources and other electronicsmay also be reduced in power or turned off for increased safety.

In accordance with still other embodiments, optionally, additionaleye-tracking cameras may be provided on the eyewear or headwearplatform, e.g., to simultaneously track responses in the region of botheyes. In healthy individuals, the reaction times of components of botheyes should be approximately the same. If the reaction times of the leftand right eyes of an individual are divergent, measurements may be usedto help diagnose physiological abnormalities or disease. For example,delayed responses in both eyes may be indicative of a central nervoussystem cognitive or systemic disorder whereas a delayed response in asingle eye or eyelid may be indicative of a peripheral neuro-muscularanomaly.

In accordance with another embodiment, wireless transmission may beadded to the eyewear or headwear to transmit information to and fromremote sites. For example, images and measurements may be transmitted toremote computing devices for further analysis and display. Conversely,wireless signals indicating reference times and/or magnitudes oftriggering events to be associated with measured responses of the head,eye, eyelid and/or pupil may be transmitted to the eyewear or headwearplatform.

In accordance with yet another embodiment, a system is provided fordetermining reaction times, magnitudes of responses, and/or otherresponses that includes a device configured to be worn on a wearer'shead; an eye-tracking camera mounted on the device oriented towards afirst eye of the wearer for capturing eye-tracking images of the firsteye; and one or more processors coupled to the eye-tracking camera foridentifying anatomical structures of the first eye within theeye-tracking camera images. The one or more processors may be configuredfor identifying reference times of external events and analyzing theeye-tracking images after the reference times to determine responses ofthe first eye, eyelid, and pupil to the external events. For example,processor(s) may identify two or more parameters selected from: a)location of at least one of the pupil and iris in one or moredimensions, b) a size of the pupil, c) a shape of the pupil, and d) alocation of the eyelid from the eye-tracking images, and detect changesin the two or more parameters to determine reaction times or otherresponses of the wearer to external events.

Optionally, the system may include one or more devices for detectingexternal events to provide reference times. For example, in oneembodiment, a scene camera may be mounted on the device oriented awayfrom the wearer for viewing an environment of the wearer and capturingscene camera images of the environment. The processor(s) may identifyreference times from changes detected in the scene camera images. Forexample, the processor(s) may use object recognition algorithms toidentify locations of objects within a field-of-view of the scene camerafrom the scene camera images, and identify reference times based on thetimes when predetermined objects are present at or absent frompredetermined locations in the scene camera images. In addition oralternatively, the processor(s) may analyze the scene camera images forscene movements to determine movements of the wearer's head, andidentify particular head movements with reference times.

In another embodiment, the system may include a detector for monitoringsounds or other activities within the wearer's environment. For example,a microphone may be mounted on the device for detecting sounds, and theprocessor(s) may be coupled to the microphone for monitoring thewearer's environment to identify predetermined sounds and associatingreference times with the predetermined sounds.

In accordance with still another embodiment, a system is provided fordetermining reaction times, magnitudes of responses, and/or otherresponses that includes a device configured to be worn on a wearer'shead; an eye-tracking camera mounted on the device oriented towards afirst eye of the wearer for capturing eye-tracking images of the firsteye; and a scene camera mounted on the device oriented away from thewearer for viewing an environment of the wearer and capturing scenecamera images of the environment. One or more processors may be coupledto the eye-tracking camera for identifying anatomical structures of thefirst eye in the eye-tracking images, and/or to the scene camera foridentifying predetermined events in the environment of the wearer fromthe scene camera images. For example, the one or more processors mayidentify a reference time when a predetermined event is identified fromthe scene camera images, and may monitor changes in one or more of theanatomical features identified in the eye-tracking images to determine areaction time or other response of the wearer to the predeterminedevent.

In an exemplary embodiment, the processor(s) may monitor the scenecamera images to identify an object in the environment of the wearerfrom the scene camera images, and may monitor the object to identifywhen the object is present at or absent from a predetermined location inthe scene camera images. In this example, the reference time may be thetime when the processor(s) identify that the object is present at orabsent from the predetermined location. In another exemplary embodiment,the system may include an electronic object including a display, and theprocessor(s) may monitor the scene camera images to identify when apredetermined image is presented on the display. In this example, thereference time may be the initial time the predetermined image isdisplayed.

In accordance with yet another embodiment, a system is provided fordetermining reaction times or other responses that includes a deviceconfigured to be worn on a wearer's head; an eye-tracking camera mountedon the device oriented towards a first eye of the wearer for capturingeye-tracking images of the first eye; and a microphone for detectingsounds within an environment of the wearer. One or more processors maybe coupled to the eye-tracking camera for identifying anatomicalstructures of the first eye in the eye-tracking images, and to themicrophone for detecting predetermined sounds generated within theenvironment of the wearer. For example, the processor(s) may identify areference time when a predetermined sound is detected and monitorchanges in one or more of the anatomical features identified in theeye-tracking images to determine a reaction time or other response ofthe wearer to the predetermined sound.

In accordance with still another embodiment, a system is provided fordetermining reaction times or other responses that includes a deviceconfigured to be worn on a wearer's head; an eye-tracking camera mountedon the device oriented towards a first eye of the wearer for capturingeye-tracking images of the first eye; and an emitter for emitting adetectable signal towards the wearer; and a sensor mounted on the devicefor providing an orientation of the wearer's head. One or moreprocessors may be coupled to the eye-tracking camera for identifyinganatomical structures of the first eye in the eye-tracking images, andto the sensor for monitoring the orientation of the wearer's head.

The processor(s) may identify a reference time when the emitter isactivated and monitor changes in the one or more of the anatomicalfeatures identified in the eye-tracking images after the reference timeto determine a reaction time or other response of the first eye. Inaddition, the processor(s) may monitor changes in the orientation of thewearer's head after the reference time to determine a reaction time orother response of the wearer's head. Optionally, the processor(s) mayanalyze one or more of the determined reaction times to determineinformation regarding the wearer. For example, the reaction time of thewearer's head may be compared to the reaction time of the first eye todetermine at least one of a physical or mental state of the wearer.

In an exemplary embodiment, the emitter may include one or more lightsources mounted on the device for emitting light towards the first eyewhen activated. In another exemplary embodiment, the emitter may includeone or more speakers mounted on the device for generating predeterminedsounds.

Optionally, the one or more processors may include a controller coupledto the emitter for activating the emitter. For example, the controllermay be configured for intermittently activating the emitter, therebyproviding a series of reference times. The processor(s) may compare thereaction time or other response of the wearer's head to the reactiontime or other response of the first eye after each of the referencetimes, e.g., to determine at least one of a physical or mental state ofthe wearer. In one embodiment, the controller may periodically orrandomly activate the emitter. In another embodiment, the system mayinclude a receiver on the device coupled to the controller, and thecontroller may receive commands from a remote location, e.g.,instructing the controller to activate the emitter based on the commandsreceived by the receiver.

In accordance with yet another embodiment, a method is provided fordetermining reaction times or other responses that includes placing adevice on a wearer's head, the device comprising a scene camera orientedaway from the wearer and an eye-tracking camera oriented towards a firsteye of the wearer; monitoring eye-tracking images from the eye-trackingcamera to identify anatomical structures of the first eye; andmonitoring scene camera images from the scene camera to identifypredetermined events in the environment of the wearer. A reference timemay be identified, e.g., when a predetermined event is identified fromthe scene camera images, and changes in one or more of the anatomicalfeatures identified in the eye-tracking images may be monitored todetermine a reaction time or other response of the wearer to thepredetermined event.

In accordance with still another embodiment, a method is provided fordetermining reaction times or other responses that includes placing adevice on a wearer's head, the device comprising an eye-tracking cameraoriented towards at least one of the wearer's eyes; monitoringeye-tracking images from the eye-tracking camera to identify anatomicalstructures of the first eye; and monitoring sounds within an environmentof the wearer to identify predetermined events within the environment. Areference time may be identified, e.g., when a predetermined sound isdetected within the environment, and changes in one or more of theanatomical features identified in the eye-tracking images may bemonitored to determine a reaction time or other response of the wearerto the predetermined sound.

In accordance with another embodiment, a method is provided fordetermining reaction times or other responses that includes placing adevice on a wearer's head, the device comprising an eye-tracking cameraoriented towards at least one of the wearer's eyes; monitoringeye-tracking images from the eye-tracking camera to identify anatomicalstructures of the first eye; and monitoring an orientation of thewearer's head. At least one of the wearer and the wearer's environmentmay be monitored, e.g., to identify predetermined events, and areference time may be identified when a predetermined event is detected.Changes in one or more of the anatomical features identified in theeye-tracking images after the reference time may be monitored todetermine a reaction time or other response of the wearer to thepredetermined event, and/or changes in the orientation of the wearer'shead after the reference time may be monitored to determine a reactiontime or other response of the wearer's head to the predetermined event.For example, the reaction times or other responses may be compared,e.g., to determine at least one of a physical or mental state of thewearer.

In accordance with yet another embodiment, a method is provided fordetermining reaction times or other responses that includes placing adevice on a wearer's head, the device comprising an eye-tracking cameraoriented towards at least one of the wearer's eyes, and monitoringeye-tracking images from the eye-tracking camera to identify anatomicalstructures of the first eye. A detectable signal, e.g., a sound orlight, may be emitted towards the wearer at a reference time, andchanges in one or more of the anatomical features identified in theeye-tracking images after the reference time may be monitored todetermine a reaction time or other response of the wearer to thedetectable signal. In addition or alternatively, changes in theorientation of the wearer's head after the reference time may bemonitored to determine a reaction time or other response of the wearer'shead to the detectable signal, e.g., to determine the wearer's physicaland/or mental state.

Other aspects and features of the present invention will become apparentfrom consideration of the following description taken in conjunctionwith the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings illustrate exemplary embodiments of the invention, inwhich:

FIG. 1 is a perspective view of an example of a system mounted oneyeglasses frames for measuring reaction times of a wearer's head, eyes,eyelids, and/or pupils.

FIG. 2 is a partial cut-away, side view of the system of FIG. 1, showingconnections among a processing unit, accelerometer, scene camera,eye-tracking camera, and other components.

FIG. 3 is an example of measured responses and reaction times of awearer's eye movement, head movement, and pupillary constriction inresponse to a flash of light.

FIG. 4A is an example of camera images gathered by a scene cameratracking an object (i.e., a baseball), and FIG. 4B is a graph showing anexemplary method to identify a reference time as the object passesthrough a selected scene reference location.

FIG. 5 is an example of measured reaction times of a wearer's eyelid,eye, and head in response to a sharp sound.

FIGS. 6A-6C show an example of a wearer's eye movement and pupillaryreaction times resulting from the display of an image of a virtualobject on a display monitor.

FIGS. 7A and 7B show an example of repeatedly measuring a wearer's eyemovement reaction times in response to following a virtual object (i.e.,a cursor) as it moves about on a display monitor.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS

Turning to the drawings, FIG. 1 shows an exemplary embodiment of asystem 10 including a set of eyeglass frames 11 with a scene camera 12,and two eye-tracking cameras 13 a, 13 b. The scene camera 12 is orientedon the frame 11 to view an area away from the wearer's head 14 in orderto track the environment 15 in the region of the device wearer. Forexample, the scene camera 12 may be used to identify the movement ofobjects, such as a football 16, or any identifiable change in anobject's or camera sub-region's size, brightness, geometry, color, andthe like, e.g., in order to establish reference times for reaction timemeasurements, as described further below. The eye-tracking cameras 13 aand 13 b are oriented on the frame 11 toward the wearer's head 14 totrack the locations of the wearer's eyelids, pupils, and/or otherreference points in one or more eyes, such as the pupil, iris, or sclera(not shown).

In this embodiment, a single processing unit 17 may be carried by theframe 11 to acquire images from the scene camera 12 as well as theeye-tracking cameras 13 a, 13 b, although it will be appreciated thatseparate processors (not shown) may be provided on the frame 11 or at aremote location (not shown) that communicates with the frame 11. Aself-contained power source (e.g., battery) 18 may be carried by theframe 11, e.g., encased in the stem of frame 11 opposite that containingprocessing unit 17.

In an exemplary embodiment, each of the scene camera 12 and eye-trackingcameras 13 a, 13 b may include a CCD or CMOS or other detector, e.g.,including a rectangular or other active area array of pixels, forcapturing images and generating video signals representing the images.The active area of the cameras 12, 13 a, 13 b may have any desiredshape, e.g., a square or rectangular shape, and the like. In addition,the cameras 12, 13 a, 13 b may include one or more filters, lenses, andthe like, if desired, e.g., to focus images on the active area, filterundesired intensities and/or wavelengths of light, and the like.

In the embodiment illustrated in FIG. 1, the scene camera 12 isunobtrusively located on the nose bridge of the frame 11, therebyminimizing interference with the wearer's normal vision. It is to beunderstood that multiple scene cameras may be included, pointed in anydirections including the back, top, and sides of a wearer's head 14. Forexample, two (2) scene cameras 19 a, 19 b may be mounted at locationsnear the outer edges of the frame 11, e.g., in addition to or instead ofthe scene camera 12. In addition or alternatively, scene cameras may belocated on the frame 11, e.g., to view an environment to the sidesand/or to the rear of the device wearer (not shown). In the case ofheadwear, a scene camera may, for example, be located atop the wearer'shead 14. Finally, fully or partially reflective surfaces that arestationary or moveable (not shown) may also be used to direct views indifferent directions to the active areas of one or more scene cameras.

It is to be understood that multiple scene cameras may be provided thatare spaced apart from one another and/or directed toward multipleregions around the device wearer to provide separate or overlappingfields-of-view. Multiple scene cameras may provide higher spatialresolutions, increased sensitivity under different lighting conditions,and/or a wider field-of-view, e.g., in addition to or instead of scenecamera 12. For example, multiple scene cameras may be used to detectobjects (e.g., projectiles) approaching the device wearer from anydirection. In addition or alternatively, another potential advantage ofusing multiple scene cameras is the ability to use different opticalfilters with each camera to isolate objects of interest that differ incolor or that are preferentially illuminated using different wavelengthsof electromagnetic radiation.

FIG. 2 shows a cut-away view and back side of the system 10 illustratedin FIG. 1. The fixed spatial displacement between the scene camera 12and eye-tracking camera 13 b mounted within frame 11 in X, Y and Zdirections may be readily seen from this perspective. This displacementmay be used in calculations to determine eye gaze-tracking vectorssimilar to the embodiments described in the references incorporated byreference elsewhere herein. For example, gaze-tracking during reactiontime measurements may be used to determine whether the device wearerlooks toward or away from events in the environment 15 (FIG. 1) thattrigger reactions. Looking toward versus looking away from an object orimage may, for example, be used to assess whether the wearer isattracted by a scene or whether there is avoidance of particular typesof visual images.

FIG. 2 also shows an example of a location where a processing unit 17for scene analysis and eye-tracking may be embedded within the stem 24of the frame 11. In this exemplary embodiment, a single processing unit17 is in the form of a field-programmable gate array (FPGA), althoughalternatively, the processing unit 17 may be an application-specificintegrated circuit (ASIC) or other processing device.

The processing unit 17 may include one or more controllers orprocessors, e.g., one or more hardware components and/or softwaremodules for operating various components of the system 10. For example,the processing unit 17 may include a separate (not shown) or integralcontroller for controlling light sources or cameras 12, 13 b, forreceiving and/or processing signals from cameras 12, 13 b, for receivingand/or processing signals from accelerometers 28, for wirelesscommunication, and the like. Optionally, one or more components of theprocessing unit 17 may be carried on the stem 24, on the lens supportsof the frame 11, nose bridge, and/or other locations within the eyewearor headwear, similar to the embodiments described in the referencesincorporated by reference elsewhere herein. In the exemplary embodimentshown in FIG. 2, a single processing unit 17 is used for imageacquisition and processing for both scene identification and eyetracking functions, as well as other sensors and electronics, forreaction time measurements.

Cable(s) 26 may include individual cables or sets of wires coupled tothe cameras 12, 13 b, battery 18 (FIG. 1), light sources, and/or othercomponents on the frame 11 and to the processing unit 17. For example,individual cables or sets of wires (not shown) may be embedded in theframe 11, e.g., along the rim from the cameras 12, 13 b, and the like,until captured within the cable 26, e.g., to reduce the overall profileof the frame 11 and/or to direct signals around any hinged regions orcorners 27 within the eyewear or headwear, as desired. Embedded cablesand miniature electronics may reduce the obtrusiveness of the eyewear orheadwear (e.g., reducing weight, size, and/or blockage of the potentialvisual field of the device wearer).

The processing unit 17 may also include memory (not shown) for storingimage signals from the camera(s) 12, 13 b, measurements of filters forediting and/or processing the image signals, and the like. Optionally,the frame 11 and/or processing unit 17 may include one or moretransmitters and/or receivers (not shown) for transmitting data,receiving instructions, and the like. In addition or alternatively, atleast some processing may be performed by components that are remotefrom the on-board processing unit 17 and frame 11, similar toembodiments disclosed in the references incorporated by referenceelsewhere herein. For example, the system 10 may include one or morereceivers, processors, and/or displays (not shown) at a remote locationfrom the processing unit 17 and/or frame 11, e.g., in the same room, ata nearby monitoring station, or at a more distant location. Suchdisplays may include views generated by the scene camera 12 oreye-tracking camera(s) 13 b, and/or graphs or tables of measurements.

Accelerometer 28 may be provided on the frame 11, e.g., located near theprocessing unit 17 or elsewhere within the eyewear or headwear. Amulti-axis accelerometer that detects accelerations in up to three (3)orthogonal directions may be used to detect movements of the wearer'shead 14 (shown in FIG. 1) as well as overall wearer's head orientationrelative to the gravitational field of the earth. The accelerometer 28may be tuned to the g-forces normally encountered during movements ofthe wearer's head 14 (e.g., less than two (2) G's). It is to beunderstood that special purpose designs may be developed for specificsituations. For example, higher sensitivity may be a requirement forinfants or small animals; whereas a greater range might be required tomonitor head movements of military fighter pilots where instantaneousforces might exceed nine (9) G's.

With additional reference to FIG. 2, a connector 29 may optionally beprovided on the frame 11, e.g., to acquire electronic signals withreference time information from devices coupled to the system 10 by theconnector 29. For example, a microphone (not shown) may be connected tothe connector 29 to provide an input signal, and the reaction times orother responses (e.g., magnitude of response over time) of the wearer'seyes, eyelids, head, and/or pupils may be measured in response to thegeneration of a sound or a series of sounds. Alternatively, one or moremicrophones (not shown) may be mounted on the frame 11 that are coupledto the processing unit 17, e.g., for substantially continuously orintermittently monitoring the wearer's environment or the wearer forpredetermined sounds. Reference times may, for example, be registeredwhen the sound level reaches a threshold value. In other embodiments,waveform recognition techniques may be used to measure reaction times inresponse to specific words or phrases. For example, the microphone(s)may be used to listen to the wearer, and the processing unit 17 maydetect when the wearer utters predetermined words or sounds, and thenmeasure the reaction time or other response of the wearer after thereference time associated with the detected event. In addition oralternatively, the processing unit 17 may monitor the microphone signalsto detect when a predetermined sound is generated within the wearer'senvironment, and the reaction time or other response of the wearer maydetermined from the reference time associated with the detected event.

Other substantially continuous or intermittent analog signals may bealso used, for example, as generated by a position sensor, strain gauge,air flow sensor, thermometer, and the like, e.g., where a temporalreference is produced when the signal reaches a chosen threshold voltageor current. For example, a temporal reference may also be produced whena particular series of voltages or currents, or waveform is encountered.In addition or alternatively, digital signals with low-to-high and/orhigh-to-low transitions, i.e., binary transitions, may also be used asreference time inputs.

With additional reference to FIG. 2, optionally, one or morelight-sources 25 may be included within the eyewear or headwear toproduce a burst of light at one or more chosen times, for example, toevoke pupillary responses and/or measure reaction times and responses ofthe device wearer. In this exemplary embodiment, the reference time(i.e., activating the light source) may be controlled by a processingunit that controls the light source and the light source 25 may be alight-emitting diode (LED).

Multiple lights sources may also be included that may be positioned atvarious locations around the frame 11 of the eyewear or headwear.Multiple light sources with similar illumination characteristic may beused to provide a more wide-spread illumination of the wearer's eye toevoke responses. Multiple light sources may also be used to evokeresponses in one eye individually or both eyes simultaneously. Inaddition or alternatively, multiple light sources with dissimilarproperties may also be used, for example, to illuminate eyes withdifferent wavelengths of light. Wavelengths of electromagnetic radiationmay be in the visible spectrum or outside the visible spectrum.

The current or voltage used to control the one or more light sources 25may also be varied in amplitude or duration to control the intensity ofillumination. For example, varying the intensity of light sources may beused to determine threshold intensities that evoke pupillary responses(e.g., at selected wavelengths). In addition or alternatively, one ormore speakers (not shown) may be provided, e.g., on the device orseparate from the device, which may generate predetermined soundstogether with or separate from activation of the light source(s) 25.

Turning to FIG. 3, an exemplary sequence of events is shown, which maybe used to measure reaction times of the wearer's eye movement, headmovement, and/or pupil size in response to a flash of light. In thisexample, light is generated by a light emitting diode 30 c at a chosentime 30 a, thereby providing a reference time that may be compared tosubsequently detected responses. For example, as shown in FIG. 2, one ormore light sources 25 used to evoke responses may be mounted, ifdesired, on the eyewear or headwear such that light emitted by the lightsource(s) 25 are directed towards the wearer's eye(s). In addition oralternatively, light sources may also be provided separate from orexternal to the device wearer. Under these conditions, the time ofoccurrence of flashes of light and light intensity 30 b may be measuredby one or more scene cameras, if desired, e.g., to determine thereference time for detecting and/or monitoring one or more responses bythe wearer.

FIG. 3 illustrates an example of device wearer eye movement 31 c, e.g.,where one or more eye-tracking cameras 13 b (shown in FIG. 2) may beused to track one or more reference locations on the wearer's eye (e.g.,edges of a pupil or iris, center of a pupil or iris, locations ofregions of the sclera, and the like), e.g. to monitor eye movement 31 cin one or two dimensions. Eye location trace 31 b represents movement inone such dimension, e.g., along a vertical or “y” axis (shown in FIG.2). As shown, signification eye movement begins to occur at a time 31 afollowing the activation of the light source 30 a. The temporaldifference between activating the light source at time 30 a anddetecting significant movement of the wearer's eye at time 31 a is thereaction time or response time 32 of the wearer's eye, i.e., thecomponents of the eyeball, such as the pupil and iris, identified fromthe eye-tracking camera images. In addition, as shown, the magnitude ofresponse of the wearer's eye, e.g., the location in one or moredimensions, may be measured over time, e.g., after time 31 a when aninitial reaction is detected.

FIG. 3 also illustrates an example of monitoring and/or identifyingwearer head movement 33 c, e.g., where a multi-axial accelerometer 28(shown in FIG. 2) may be used to measure relative displacements of thehead 14 (FIG. 1) in one or more dimensions. Head displacement trace 33 brepresents movement in one such dimension, e.g., within a horizontalplane and/or orthogonal to a vertical or “y” axis (shown in FIG. 2). Asshown, signification head movement begins to occur at a time 33 afollowing the activation of a light source at time 30 a. The temporaldifference between activating the light source 30 a and detectingsignificant movement of the wearer's head at time 33 a is the reactiontime 34 of the wearer's head. Similar to the eye 31, in addition, ifdesired, the magnitude of response of the wearer's head, e.g., degree ofturning, speed of turning, and the like, may be measured over time,e.g., after time 33 a when an initial reaction is detected.

FIG. 3 also illustrates a pupillary response 35 c to increased lightlevels, e.g., where one or more eye-tracking cameras 13 b (shown in FIG.2) may be used to track the size and/or geometry of one or both of thedevice wearer's pupils. Pupil size trace 35 b represents changes in thearea occupied by one pupil as measured by an eye-tracking camera 13 band associated processing unit 17 (shown in FIG. 2). In response toactivating the light source 30 a, the wearer's pupil begins to constrictat time 35 a and continues to be constricted for as long as the lightremains on. The temporal difference between activating the light sourceat time 30 a and the onset of significant pupillary constriction at time35 a is the wearer's pupillary reaction time 36. In addition, themagnitude of response of the wearer's pupil, e.g., the size and/or shapeof the pupil, may be measured over time, e.g., after time 35 a when aninitial reaction is detected.

Turning to FIG. 4, exemplary methods are shown for using a scene camera12 (e.g., as shown in FIG. 1) to generate one or more temporal referencepoints based on the movement of an object within a scene. A series ofimages 40 a, 40 b, 40 c, 40 d are shown that are selected from a videosequence gathered using a scene camera 12 (shown in FIG. 1). An object,represented in this case as a baseball 41, passes through the scene andmay be identified and/or localized in images acquired by the scenecamera 12, e.g., using machine vision techniques that are known in theart. For example, the processing unit 17 (e.g., shown in FIG. 1) mayinclude or otherwise access a database of known templates, e.g., a tableassociating known objects with data identifying their shapes and/orcolors. The database may include reference points of known objects,detailed color and/or shape information on the reference objects, andthe like, mapped to particular physical objects, thereby providing theprocessing unit sufficient information to identify the encounteredobject, such as the baseball 41. When the object reaches a selectedreference location or region 42 in the camera images 40 a, 40 b, 40 c,40 d, a reference time is registered.

With additional reference to FIG. 4, it is possible to visualize thepresence or absence of an object 41 at a selected location 42 in acamera image using a time trace 43. In this example, a low state 46 inthe time trace represents the absence of an identifiable object at theselected location 42. Conversely, a high state 47 represents thepresence of an identifiable object at the selected location or region42. A low-to-high transition 44 provides a temporal reference point forthe arrival of the object at the location. A high-to-low transition 45may also be used as a temporal reference point, if desired, indicatingthe departure of the object. Similar strategies may be used to trackmultiple objects, traveling in multiple directions, traversing multiplereference locations or regions, and/or in multiple orientations withinimages generated by one or more scene cameras.

Turning to FIG. 5, additional exemplary methods are shown for measuringresponse times based on audible references, e.g., following theoccurrence of a sharp sound. The system 10 may include a microphone (notshown), e.g., mounted on the frame 12 or elsewhere on eyewear orheadwear, or positioned remotely with signals supplied to or orientedtowards the eyewear or headwear and/or wearer, may be used to generate asound represented by sound time trace 50 b. The occurrence of a soundmay be detected as the presence of vibrations (i.e., generated bytraveling compression waves in the air and sensed by a microphone) onthe trace 50 b, e.g., where the onset of a sharp sound 50 a may be usedas a temporal reference point. Particular waveforms may also beidentified to produce multiple distinct and/or periodic temporalreference points. For example, sounds associated with the formation ofwords may be identified using speech recognitions techniques that areknown in the art, e.g., to identify particular words and trigger one ormore temporal reference points.

One or more eye-tracking cameras 13 b (FIG. 2) mounted on the eyewear orheadwear may be used to track movement of one or both of the wearer'seyelids in order to generate an eyelid position trace 51 b. In thiscase, closure of one eyelid is represented as a downward deflection inthe eyelid position trace 51 b starting at time 51 a. In FIG. 5, thetemporal difference between the occurrence of the sound, e.g.,activation of the microphone at time 50 a, and significant movement ofthe eyelid at time 51 a is the reaction time 52 of the eyelid.

The same (or different) eye-tracking camera(s) mounted on the eyewear orheadwear (e.g., cameras 13 a, 13 b shown in FIG. 1) may be used to trackmovement of reference locations of one or both wearer's eyes (e.g.,edges of pupils, structures with the iris, and the like, similar toother embodiments herein), e.g., in order to generate one ormultiple-dimensional eye location traces. With additional reference toFIG. 5, a representative trace is shown indicating the position in onedimension of one eye, e.g., along a vertical or “y” axis, as an eyelocation time trace 53 b. The temporal difference between the occurrenceof a sound 50 a and significant movement of the eye at time 53 a is thereaction time 54 of the eye. In this particular example, closure of theeyelid, as depicted in trace 51 b, over the iris and pupil produces abrief time during which eye movement cannot be monitored, e.g., duringtime span 55, because of blockage of the view of the pupil and iriswithin eye-tracking camera images.

As described previously, a multi-dimensional accelerometer 28 (shown inFIG. 2) may be mounted on the eyewear or headwear and used to monitormovement of the head. As discussed earlier, accelerometers, for examplebased on MEMS technologies, may monitor accelerations in all three (3)orthogonal dimensions substantially simultaneously. With additionalreference to FIG. 5, a representative trace indicating the movement ofthe head in a selected dimension is illustrated as a head location timetrace 56 b. The temporal difference between the occurrence of the soundat time 50 a and significant movement of the head at time 56 a is thereaction or response time 57 of the head.

Turning to FIGS. 6A-6C, another exemplary method is shown for measuringthe reaction times (and/or optionally other responses) of structures ofthe eye in response to the display of one or more virtual objects, e.g.,presented via a picture, book, computer monitor, mobile computingdevice, television, theater screen, billboard, and the like. In thisexample, as shown in FIGS. 6A and 6B, a system may be provided thatincludes a eyewear or headwear device for monitoring a user or wearer(not shown, e.g., the system 10 shown in FIG. 1) and a system includinga computer display monitor 60 a, 60 b for displaying images to thewearer. As shown in FIG. 6B, the monitor 60 b may be used to depict theoccurrence of an explosion or other sudden image 61, e.g., replacingother previous images (not shown) presented on the monitor 60 a, asshown in FIG. 6A. The time that the initial image of the explosion 61 isdisplayed on the monitor 60 b may be recorded or otherwise known by thesystem as a reference time, e.g., provided by a computer, processor, orother electronic device coupled to the monitor 60 for generating theimage(s). The reference time may be transmitted to a processor, e.g., inframe 12 of the system 10 (shown in FIG. 1) or elsewhere, or imagerecognition from scene images using one or more scene cameras 12 (e.g.,shown in FIG. 1) may be used to measure the time of first appearance ofthe sudden image 61 to register a reference time. The explosion displayreference time is depicted as a low-to-high transition 66 a on imagedisplay time trace 66 b of FIG. 6C.

FIGS. 6A-6C further illustrate a reaction by a wearer's eye 62 to thedisplay of the virtual explosion 61. For example, prior to the displayof the virtual explosion 61, an eye-tracking camera 13 a, 13 b (e.g.,shown in FIG. 1) may acquire images that show the eye 62 a with a pupil63 a of typical size, e.g., for room-lit or other ambient conditions.The pupil 63 a is surrounded by an iris 64 which, in turn, is surroundedby the “white of the eye” or sclera 65. Following the display of theexplosion 61, measurements of the eye 62 b from images acquired by theeye-tracking camera may reveal that the center of the pupil 63 b hasmoved and/or that the pupil 63 b has significantly dilated. Eye movementin a selected dimension, e.g., along a vertical or “y” axis, is depictedin an eye location time trace 67 b in FIG. 6C, where the time ofoccurrence of initial eye movement in response to the virtual explosionis registered at time 67 a. The temporal difference between the displayof an explosion 61 at time 66 a and significant movement of the eye attime 67 a is the reaction or response time 67 c of the eye.

In addition or alternatively, the size of the pupil 63 may be trackedusing an eye-tracking camera 13 a, 13 b (e.g., shown in FIG. 1) anddepicted as a time trace 68 b in FIG. 6C, where the time of onset ofpupil dilation in response to a virtual explosion 61 is marked 68 a. Thetemporal difference between the display of an explosion 61 at time 66 aand significant pupil dilation at time 68 a is the pupillary reaction orresponse time 68 c. Tracking the combination of whether a response tothe display of an image causes attraction (i.e., focus) or aversion tothe image along with measuring the degree of pupillary dilation may, forexample, assist in the diagnosis of post traumatic stress disorderand/or to measure the effectiveness of advertising. Thus, theintegration of reaction times and/or magnitudes of multiple responses bya wearer may be monitored to more accurately diagnose, predict, and/oranalyze behavior of the wearer than monitoring eye movement or anotherparameter alone.

FIGS. 7A and 7B illustrate the ability of the systems and methodsdescribed herein to make repeated reaction time measurements. In thisexample, the device wearer of a system, such as the system 10 of FIG. 1,may be asked to visually track a virtual object, e.g., a computer cursor71, around the screen of a display monitor 70, as shown in FIG. 7A. Thedevice wearer may be instructed to follow the cursor 71 as it is steppedthrough a series of screen locations 72 a, 72 b, 72 c, 72 d, 72 e over aperiod of time. The times when cursor locations 72 a, 72 b, 72 c, 72 d,72 e are changed are known to the computer generating the display andmay be transmitted to the system 10, or image recognition using one ormore scene cameras 12 (e.g., as shown in FIG. 1) may be used to measureshifts in cursor location to generate a series of reference times.

With additional reference to FIG. 7A, where images are displayed on atwo-dimensional screen 70, cursor locations 72 a, 72 b, 72 c, 72 d, 72 emay be described by a pair of orthogonal coordinates, e.g., representing“X” (e.g., horizontal) and “Y” (e.g., vertical) dimensions within theplane of the screen 70. Movements in the X direction may be used togenerate a time trace 73 for X-axis displacements, as shown in FIG. 7B.Eye movements may be tracked using an eye-tracking camera 13 a, 13 b(e.g., as shown in FIG. 1). Displacements by an eye in the X directionmay be used to generate an eye location time trace 74, also as shown inFIG. 7B. Differences between the times of occurrence of cursor movementsin the X direction and corresponding initial movements of the eye in theX direction may be used to generate a series of X-direction reactiontimes 75 a, 75 b, 75 c.

Similarly, movements in the Y direction may be used to generate an eyelocation time trace 76 for Y-axis displacements. Displacements by an eyemeasured in the Y direction may be used to generate an additional eyelocation time trace 77, as shown in FIG. 7B. Differences between thetimes of occurrence of cursor movements in the Y direction andcorresponding initial movements of the eye in the Y direction may beused to generate a series of Y-direction reaction times 78 a, 78 b, 78 cof the device wearer. Differences in reaction times in horizontal andvertical directions may indicate the presence of a muscular orneurological disorder. Changes in X- and Y-direction reaction times overa prolonged period may, for example, be used to assess fatigue.

The foregoing disclosure of the exemplary embodiments has been presentedfor purposes of illustration and description. It is not intended to beexhaustive or to limit the invention to the precise forms disclosed.Many variations and modifications of the embodiments described hereinwill be apparent to one of ordinary skill in the art in light of theabove disclosure.

Further, in describing representative embodiments, the specification mayhave presented methods and/or processes as a particular sequence ofsteps. However, to the extent that the methods or processes do not relyon the particular order of steps set forth herein, the methods orprocesses should not be limited to the particular sequence of stepsdescribed. As one of ordinary skill in the art would appreciate, othersequences of steps may be possible. Therefore, the particular order ofthe steps set forth in the specification should not be construed aslimitations on the claims.

While the invention is susceptible to various modifications andalternative forms, specific examples thereof have been shown in thedrawings and are herein described in detail. It should be understood,however, that the invention is not to be limited to the particular formsor methods disclosed, but to the contrary, the invention is to cover allmodifications, equivalents and alternatives falling within the scope ofthe appended claims.

1. A system for determining responses of a subject, comprising: a deviceconfigured to be worn on a wearer's head; an eye-tracking camera mountedon the device oriented towards a first eye of the wearer for capturingeye-tracking images of the first eye; and one or more processors coupledto the eye-tracking camera for identifying anatomical structures of thefirst eye within the eye-tracking camera images, the one or moreprocessors configured for identifying reference times of external eventsand analyzing the eye-tracking images after the reference times todetermine at least one of magnitudes of responses and reaction times ofthe first eye, eyelid, and pupil to the external events. 2-27.(canceled)
 28. A system for determining responses of a subject,comprising: a device configured to be worn on a wearer's head; aneye-tracking camera mounted on the device oriented towards a first eyeof the wearer for capturing eye-tracking images of the first eye; ascene camera mounted on the device oriented away from the wearer forviewing an environment of the wearer and capturing scene camera imagesof the environment; and one or more processors coupled to theeye-tracking camera for identifying anatomical structures of the firsteye in the eye-tracking images, and coupled to the scene camera foridentifying predetermined events in the environment of the wearer fromthe scene camera images, the one or more processors configured foridentifying a reference time when a predetermined event is identifiedfrom the scene camera images, and monitoring changes in one or more ofthe anatomical features identified in the eye-tracking images todetermine at least one of a magnitude of response and a reaction time ofthe wearer to the predetermined event. 29-48. (canceled)
 49. A methodfor determining responses of a subject, comprising: placing a device ona wearer's head, the device comprising an eye-tracking camera orientedtowards a first eye of the wearer; utilizing an input to determinereference times; determining locations of anatomical structures of theperson wearing the device within eye-tracking camera images acquired bythe eye-tracking camera; and using the reference times and the locationsdetermined from the eye-tracking camera images to determine at least oneof magnitudes of responses and reaction times of the wearer's eye,eyelid and pupil.
 50. The method of claim 49, wherein the devicecomprises a scene camera oriented away from the wearer for acquiringscene camera images of environment of the wearer.
 51. The method ofclaim 50, further comprising determining reference times from changes inthe wearer' environment detected in the scene camera images.
 52. Themethod of claim 51, wherein the input comprises using object recognitionto identify locations of objects within a field-of-view, therebyproviding the reference times. 53-55. (canceled)
 56. The method of claim51, wherein a reference time is identified based upon a presence versusabsence of an object at a selected location within the scene cameraimages. 57-58. (canceled)
 59. The method of claim 49, wherein amicrophone is used as a reference times input. 60-63. (canceled)
 64. Themethod of claim 49, further comprising activating a light source todirect light at the first eye, activation of the light source comprisinga reference time.
 65. The method of claim 64, further comprisingactivating one or more light sources to direct light at the first eye,activation of the one or more light sources comprising one or morereference times.
 66. The method of claim 65, further comprisingactivating one or more light sources to direct light at a second eye ofthe wearer.
 67. The method of claim 65, further comprising modulatingthe light source to adjust brightness levels to provide desiredbrightness levels to measure responses by the first eye. 68-69.(canceled)
 70. The method of claim 49, further comprising measuringaccelerations of the wearer's head in up to three (3) dimensions. 71.The method of claim 70, further comprising using reference times andmeasured accelerations to determine reaction times of the wearer's head.72. A method for determining responses of a subject, comprising: placinga device on a wearer's head, the device comprising a scene cameraoriented away from the wearer and an eye-tracking camera orientedtowards a first eye of the wearer; monitoring eye-tracking images fromthe eye-tracking camera to identify anatomical structures of the firsteye; monitoring scene camera images from the scene camera to identifypredetermined events in the environment of the wearer; identifying areference time when a predetermined event is identified from the scenecamera images; and monitoring changes in one or more of the anatomicalfeatures identified in the eye-tracking images to determine at least oneof a magnitude of response and a reaction time of the wearer to thepredetermined event.
 73. The method of claim 72, wherein the scenecamera images are monitored to identify an object in the environment ofthe wearer, and wherein identifying a reference time comprisesidentifying when the object is present at or absent from a predeterminedlocation in the scene camera images.
 74. The method of claim 73, whereinthe object is identified by accessing a database of templates mappingknown objects with at least one of shapes, colors, and sizes associatedwith the respective objects.
 75. The method of claim 72, furthercomprising providing an electronic device including a display, andwherein the scene camera images are monitored to identify when apredetermined image is presented on the display, the initial time thepredetermined image is displayed comprising the reference time.
 76. Themethod of claim 72, monitoring an orientation of the wearer's head afterthe reference time to determine a reaction time of the wearer's head tothe predetermined event, and comparing the reaction time of the wearer'shead to the reaction time of the wearer based on the changes in one ormore of the anatomical features identified in the eye-tracking images todetermine at least one of a physical or mental state of the wearer. 77.A method for determining responses of a subject, comprising: placing adevice on a wearer's head, the device comprising an eye-tracking cameraoriented towards at least one of the wearer's eyes; monitoringeye-tracking images from the eye-tracking camera to identify anatomicalstructures of the first eye; monitoring sounds within an environment ofthe wearer to identify predetermined events within the environment;identifying a reference time when a predetermined sound is detectedwithin the environment; and monitoring changes in one or more of theanatomical features identified in the eye-tracking images to determineat least one of a magnitude of response and a reaction time of thewearer to the predetermined sound.
 78. The method of claim 77, whereinthe sounds within the environment are monitored using a microphone onthe device. 79-85. (canceled)